Qualifying or certifying software should require formal methods. Such methods are formal fixed length or sequential length tests based on successes/ failures of individual tests that reflect how the user will exercises this software. These formal methods are referred to as formal qualification (QUAL) tests in this invention. QUAL tests usually start once the software has successfully undergone nominal, off-nominal, and stressed testing called Test, Analyze And Fix (TAAF). Both TAAF and QUAL testing need to be conducted under well-defined plans and procedures with an-agreed-to test criteria with the user. This criteria describes what constitutes success and failure, what tests should be performed: nominal, off-nominal and stress.
The present inventor has discovered, however, that QUAL tests based on classical statistical test procedures fall short and gives erroneous results. These classical test procedures, fixed length or sequential hypothesis type of tests, are based on well-defined stochastic processes such as Bernoulli or Poisson or their derivatives. The present inventor has determined that software does not behave in any well-defined stochastic fashion; but instead in an ill-defined fashion—more deterministic than stochastic. The user does not exercise software randomly; but generally uses a well-defined repeatable sequence of inputs that, in turn, executes code that has been tested, debugged and retested over and over again to ensure very predictable results. The uncertainty in the behavior of the software is due instead to when the user does something unexpected or if there is a rare combination of inputs that “hits” untested code. This uncertainty cannot be described by well-behaved stochastic processes or their derivatives; but can be described as beliefs or least-wise by subjective probabilities.
The present inventor has determined that beliefs functions, in particular Bayesian Beliefs Networks (BBN), are adequate in addressing both TAAF and QUAL test in terms of how mature the software is. Belief networks are relied on to model uncertainty in a variety of fields ranging from medical diagnosis to software engineering. Belief networks are rooted in probability theory and are based on qualitative expert assumptions about underlying conditions and findings, which make them simple yet powerful models for predicting outcomes of relevant experiments.
This invention illustrates two novel designs of experiments embodiments of Bayes Information Science (BIS) and BBN. These embodiments provide for risk statements concerning the software maturity from both TAAF and QUAL tests. The first embodiment uses BBN to permit the convolution of those subsets of TAAF tests that reflect nominal usage with the QUAL tests, which are typically nominal. For the first embodiment, in accordance with Bayesian mathematics, the BBN incorporates a priori evidence from past test data—as long it reflects nominal use cases—with the most recent test data to form an a posteriori belief distribution. The second embodiment uses both BBN and BIS to formally incorporate not only the TAAF nominal tests, but also incorporates TAAF off-nominal and stress tests as well and finally regression tests used in verifying corrective actions. These designs of experiments do not require a well-behave, stochastic process.